OCTA Repeatability & Reliability — Clean Data

Data

This data was taken from a clean dataset that Jae emailed out on November 26th, 2019.

Jae organized the retina by six different methods. He created OCTA variable scores for the overall image, 2 hemifields, 4 quadrants, 6 sectors, 12 clock hours, and an annular ring.

Annular image scoring was also assessed, and ICC’s were similar, but slightly lower than global. Results for annular image scoring are not shown. For now, only the overall “global” image scoring will be shown.

Zeiss commercial vessel area density and flux variables are also shown for the global image scoring.

Data Wrangling

Rename Variables

This section will rename the variables to match the dictionary

Label Variables

This section will label variables appropriately.

Data Dictionary

Variable	Label
ptid	Patient Identification
Sociodemographic Variables
age	Age
female	Sex
Systemic Clinical Variables
htn_dx	Hypertension Diagnosis
dm_dx	Diabetes Mellitus Diagnosis
nro_dx	Optic Neuropathy
Ophthalmic Clinical Variables
dr_dx	Diabetic Retinopathy Diagnosis, Ordinal
dr_dx_bin	Diabetic Retinopathy Diagnosis, Binary
glc_sus_dx	Glaucoma Suspect Diagnosis
glc_dx	Glaucoma Diagnosis, Ordinal
glc_dx_bin	Glaucoma Diagnosis, Binary
surg_proc_glaucoma	Surgical Procedure for Glaucoma
OCT Angiography Variables
surg_proc_catextract	Surgical Procedure for Cataract Extraction
eyeid	Eye Identification
eye_imaged	Eye Imaged
time_stamp	Exam Date and Time
name_last	Last Name
name_first	First Name
image_area	Image Area
image_method	Image Method
image_size	Image Size
signal_strength	Signal Strength
v_diameter	Vessel Diameter (units?)
v_area_density	Vessel Area Density (units?)
v_skeleton_density	Vessel Skeleton Density (units?)
v_perimeter_index	Vessel Perimeter Index (units?)
v_complexity_index	Vessel Complexity Index (units?)
flow_impair_zone	Flow Impairment Zone (units?)
flux	Flux (units?)
zeiss_v_area_density	Vessel Area Density (units?), Zeiss Commercial
zeiss_flux	Flux (units?), Zeiss Commercial

Single Eye Participation

Venugopal (2019) et al. ran the analysis including all eyes.¹ Then they re-ran the analysis after selecting only a single eye randomly from each individual. They conclude minimal difference in the findings, and they report only the analysis that includes all eyes:

“The entire analysis was repeated considering one eye per subject (one eye was randomly chosen from subjects contributing both eyes for the primary analysis) and all the results were similar to the primary analysis.”¹

The following results are for the single eye, chosen at random, from each participant. Bruce did explain a way to calculate ICC’s for all measurements on both eyes of each participant. This will be coded later.

Subsetting Cohorts

Intra-Visit Data

Intra-visit data was restricted to all measurements of the same eye that occurred on the same day. Some people had multiple measurements on multiple days. Their first day was included only.

Inter-Visit Data

Inter-visit data was restricted to the first measurement of the day for each eye. If a patient had their right eye measured:

• 3 times on Monday

• 5 times on Wednesday, and

• 2 times on Friday,

Then they would have three observations in this dataset:

• The first measurement from Monday,

• The first measurement from Wednesday, and

• The first measurement from Friday.

Data Flow Chart

This tallies how many participants, how many eyes, and how many measurements are included in the two datasets.

dataset	n_participants	n_eyes	n_measurements
Global Data
Intra-Visit	88	88	214
Inter-Visit	88	88	241

Table 1: Summary Statistics

Intra-Visit OCTA Summary Statistics

Variables in intra-visit dataset

Participants

	None	Glaucoma	p	test
n	29	59
Sociodemographic Variables
age (mean (SD))	52.31 (17.08)	60.15 (15.12)	0.031
Sex = Male (%)	13 (44.8)	30 ( 50.8)	0.761
Systemic Clinical Variables
Hypertension Diagnosis = Yes (%)	9 (31.0)	21 ( 35.6)	0.853
Diabetes Mellitus Diagnosis = Yes (%)	2 ( 6.9)	11 ( 18.6)	0.254
Optic Neuropathy = Yes (%)	1 ( 3.4)	0 ( 0.0)	0.715
Ophthalmic Clinical Variables
Glaucoma Diagnosis, Binary = Glaucoma (%)	0 ( 0.0)	59 (100.0)	<0.001
Diabetic Retinopathy Diagnosis, Binary = Yes (%)	0 ( 0.0)	4 ( 6.8)	0.373
Ophthalmic Surgical Variables
Surgical Procedure for Cataract Extraction = Yes (%)	5 (17.2)	13 ( 22.0)	0.808
Surgical Procedure for Glaucoma (%)			0.004
None	25 (86.2)	31 ( 52.5)
Laser	4 (13.8)	8 ( 13.6)
Minimally Invasive Glaucoma Surgery	0 ( 0.0)	1 ( 1.7)
Filter	0 ( 0.0)	19 ( 32.2)
OCT Angiography Signal Strength
signal_strength (mean (SD))	9.29 (0.71)	9.11 (0.75)	0.278

Measurements

	Normal	Glaucoma	p	test
n	73	141
Sociodemographic Variables
age (mean (SD))	52.04 (17.18)	59.48 (15.10)	0.001
Sex = Male (%)	34 ( 46.6)	73 ( 51.8)	0.564
Systemic Clinical Variables
Hypertension Diagnosis = Yes (%)	23 ( 31.5)	50 ( 35.5)	0.670
Diabetes Mellitus Diagnosis = Yes (%)	4 ( 5.5)	24 ( 17.0)	0.031
Optic Neuropathy = Yes (%)	3 ( 4.1)	0 ( 0.0)	0.070
Diabetic Retinopathy Diagnosis
Diabetic Retinopathy Diagnosis, Ordinal (%)			NaN
None	73 (100.0)	132 ( 93.6)
Mild Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	5 ( 3.5)
Moderate Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	2 ( 1.4)
Severe Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	2 ( 1.4)
Proliferative Diabetic Retinopathy	0 ( 0.0)	0 ( 0.0)
Glaucoma Suspect Diagnosis
Glaucoma Suspect Diagnosis (%)			<0.001
None	3 ( 4.1)	141 (100.0)
Mild	25 ( 34.2)	0 ( 0.0)
Moderate	22 ( 30.1)	0 ( 0.0)
Severe	23 ( 31.5)	0 ( 0.0)
Glaucoma Diagnosis, Ordinal
Glaucoma Diagnosis, Ordinal (%)			<0.001
None	73 (100.0)	0 ( 0.0)
Mild	0 ( 0.0)	53 ( 37.6)
Moderate	0 ( 0.0)	38 ( 27.0)
Severe	0 ( 0.0)	50 ( 35.5)
Surgical Procedure for Glaucoma
Surgical Procedure for Glaucoma (%)			<0.001
None	63 ( 86.3)	76 ( 53.9)
Laser	10 ( 13.7)	18 ( 12.8)
Minimally Invasive Glaucoma Surgery	0 ( 0.0)	3 ( 2.1)
Filter	0 ( 0.0)	44 ( 31.2)
Surgical Procedure for Cataract Extraction
Surgical Procedure for Cataract Extraction = Yes (%)	12 ( 16.4)	29 ( 20.6)	0.586
Image Specifications
Eye Imaged = Right (%)	39 ( 53.4)	73 ( 51.8)	0.932
Image Area = ONH (%)	73 (100.0)	141 (100.0)	NA
Image Method = Angiography (%)	73 (100.0)	141 (100.0)	NA
Image Size = 6x6 (%)	73 (100.0)	141 (100.0)	NA
OCT Angiography Signal Strength
Signal Strength (%)			0.320
7	0 ( 0.0)	4 ( 2.8)
8	14 ( 19.2)	31 ( 22.0)
9	23 ( 31.5)	50 ( 35.5)
10	36 ( 49.3)	56 ( 39.7)
OCT Angiography Variables
v_diameter (mean (SD))	24.99 (0.70)	26.29 (1.85)	<0.001
v_area_density (mean (SD))	0.36 (0.05)	0.29 (0.08)	<0.001
zeiss_v_area_density (mean (SD))	0.43 (0.03)	0.40 (0.04)	<0.001
v_skeleton_density (mean (SD))	0.15 (0.02)	0.11 (0.04)	<0.001
v_perimeter_index (mean (SD))	0.33 (0.05)	0.26 (0.08)	<0.001
v_complexity_index (mean (SD))	5439.74 (744.47)	4305.54 (1209.65)	<0.001
flow_impair_zone (mean (SD))	5.17 (2.73)	8.81 (4.33)	<0.001
flux (mean (SD))	0.13 (0.02)	0.10 (0.03)	<0.001
zeiss_flux (mean (SD))	0.36 (0.03)	0.32 (0.04)	<0.001

Inter-Visit OCTA Summary Statistics

Variables in inter-visit dataset

Participants

	None	Glaucoma	p	test
n	29	59
Sociodemographic Variables
age (mean (SD))	52.31 (17.08)	60.15 (15.12)	0.031
Sex = Male (%)	13 (44.8)	30 ( 50.8)	0.761
Systemic Clinical Variables
Hypertension Diagnosis = Yes (%)	9 (31.0)	21 ( 35.6)	0.853
Diabetes Mellitus Diagnosis = Yes (%)	2 ( 6.9)	11 ( 18.6)	0.254
Optic Neuropathy = Yes (%)	1 ( 3.4)	0 ( 0.0)	0.715
Ophthalmic Clinical Variables
Glaucoma Diagnosis, Binary = Glaucoma (%)	0 ( 0.0)	59 (100.0)	<0.001
Diabetic Retinopathy Diagnosis, Binary = Yes (%)	0 ( 0.0)	4 ( 6.8)	0.373
Ophthalmic Surgical Variables
Surgical Procedure for Cataract Extraction = Yes (%)	5 (17.2)	13 ( 22.0)	0.808
Surgical Procedure for Glaucoma (%)			0.004
None	25 (86.2)	31 ( 52.5)
Laser	4 (13.8)	8 ( 13.6)
Minimally Invasive Glaucoma Surgery	0 ( 0.0)	1 ( 1.7)
Filter	0 ( 0.0)	19 ( 32.2)
OCT Angiography Signal Strength
signal_strength (mean (SD))	9.35 (0.65)	9.09 (0.64)	0.081

Measurements

	Normal	Glaucoma	p	test
n	69	172
Sociodemographic Variables
age (mean (SD))	53.84 (16.48)	59.47 (14.94)	0.011
Sex = Male (%)	30 ( 43.5)	90 ( 52.3)	0.272
Systemic Clinical Variables
Hypertension Diagnosis = Yes (%)	23 ( 33.3)	65 ( 37.8)	0.616
Diabetes Mellitus Diagnosis = Yes (%)	4 ( 5.8)	33 ( 19.2)	0.016
Optic Neuropathy = Yes (%)	3 ( 4.3)	0 ( 0.0)	0.035
Diabetic Retinopathy Diagnosis
Diabetic Retinopathy Diagnosis, Ordinal (%)			NaN
None	69 (100.0)	162 ( 94.2)
Mild Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	6 ( 3.5)
Moderate Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	2 ( 1.2)
Severe Non-Proliferative Diabetic Retinopathy	0 ( 0.0)	2 ( 1.2)
Proliferative Diabetic Retinopathy	0 ( 0.0)	0 ( 0.0)
Glaucoma Suspect Diagnosis
Glaucoma Suspect Diagnosis (%)			<0.001
None	3 ( 4.3)	172 (100.0)
Mild	19 ( 27.5)	0 ( 0.0)
Moderate	22 ( 31.9)	0 ( 0.0)
Severe	25 ( 36.2)	0 ( 0.0)
Glaucoma Diagnosis, Ordinal
Glaucoma Diagnosis, Ordinal (%)			<0.001
None	69 (100.0)	0 ( 0.0)
Mild	0 ( 0.0)	61 ( 35.5)
Moderate	0 ( 0.0)	43 ( 25.0)
Severe	0 ( 0.0)	68 ( 39.5)
Surgical Procedure for Glaucoma
Surgical Procedure for Glaucoma (%)			<0.001
None	60 ( 87.0)	82 ( 47.7)
Laser	9 ( 13.0)	23 ( 13.4)
Minimally Invasive Glaucoma Surgery	0 ( 0.0)	2 ( 1.2)
Filter	0 ( 0.0)	65 ( 37.8)
Surgical Procedure for Cataract Extraction
Surgical Procedure for Cataract Extraction = Yes (%)	14 ( 20.3)	39 ( 22.7)	0.817
Image Specifications
Eye Imaged = Right (%)	36 ( 52.2)	88 ( 51.2)	1.000
Image Area = ONH (%)	69 (100.0)	172 (100.0)	NA
Image Method = Angiography (%)	69 (100.0)	172 (100.0)	NA
Image Size = 6x6 (%)	69 (100.0)	172 (100.0)	NA
OCT Angiography Signal Strength
Signal Strength (%)			0.348
7	1 ( 1.4)	7 ( 4.1)
8	12 ( 17.4)	36 ( 20.9)
9	22 ( 31.9)	64 ( 37.2)
10	34 ( 49.3)	65 ( 37.8)
OCT Angiography Variables
v_diameter (mean (SD))	25.03 (0.67)	26.42 (1.82)	<0.001
v_area_density (mean (SD))	0.36 (0.05)	0.27 (0.08)	<0.001
zeiss_v_area_density (mean (SD))	0.43 (0.02)	0.39 (0.04)	<0.001
v_skeleton_density (mean (SD))	0.14 (0.02)	0.11 (0.03)	<0.001
v_perimeter_index (mean (SD))	0.33 (0.05)	0.25 (0.07)	<0.001
v_complexity_index (mean (SD))	5375.08 (782.14)	4137.99 (1138.20)	<0.001
flow_impair_zone (mean (SD))	5.33 (2.91)	9.32 (4.20)	<0.001
flux (mean (SD))	0.13 (0.02)	0.09 (0.03)	<0.001
zeiss_flux (mean (SD))	0.35 (0.03)	0.32 (0.03)	<0.001

Exploratory Data Analysis

Scatter Plots

Scatter plots of OCTA variables by Subject ID for about 10 individuals randomly sampled. These are separated by intra- and inter-visit datasets, and are useful for understanding the data better.

Vessel Area Density

Vessel Area Density, Zeiss

Vessel Diameter

Vessel Skeleton Density

Vessel Perimeter Index

Vessel Complexity Index

Flow Impairment Zone

Flux

Flux, Zeiss

Spaghetti Plots

Spaghetti plots of OCTA variables for about 10 individuals randomly sampled.

Vessel Area Density

Vessel Area Density, Zeiss

Vessel Diameter

Vessel Skeleton Density

Vessel Perimeter Index

Vessel Complexity Index

Flow Impairment Zone

Flux

Flux, Zeiss

Bland-Altman Plots

Coding of Bland-Altman (BA) plots is explained in this blog post.

Intra-Visit Bland-Altman Plots

Intra-Visit Bland-Altman Plots of OCTA Variables

Vessel Area Density

Vessel Area Density, Zeiss

Vessel Diameter

Vessel Skeleton Density

Vessel Perimeter Index

Vessel Complexity Index

Flow Impairment Zone

Flux

Flux, Zeiss

Outliers

Assess some of the observations outside of expected limits.

eyeid	ptid	female	eye_imaged	image_area	image_size	image_method	exam_date	time_stamp	v_diameter	v_area_density	v_skeleton_density	v_perimeter_index	v_complexity_index	flow_impair_zone	flux	signal_strength	zeiss_v_area_density	zeiss_flux	age	htn_dx	dm_dx	nro_dx	dr_dx	glc_sus_dx	glc_dx	surg_proc_glaucoma	surg_proc_catextract	glc_dx_bin	dr_dx_bin	n_msr_day	order_obs
27	1787031	Female	Right	ONH	6x6	Angiography	2019-04-10	2019-04-10 11:47:25	25.55900	0.2113727	0.0826999	0.1941514	3261.600	12.6711	0.06827361	9	0.386	0.286	52	No	No	No	None	Severe	None	None	Yes	Normal	No	2	1
27	1787031	Female	Right	ONH	6x6	Angiography	2019-04-10	2019-04-10 11:47:52	24.97387	0.3639228	0.1457214	0.3344465	5332.127	4.8296	0.12722039	8	0.378	0.280	52	No	No	No	None	Severe	None	None	Yes	Normal	No	2	2

Inter-Visit Bland-Altman Plots

Inter-Visit Bland-Altman Plots of OCTA Variables

Vessel Area Density

Vessel Area Density, Zeiss

Vessel Diameter

Vessel Skeleton Density

Vessel Perimeter Index

Vessel Complexity Index

Flow Impairment Zone

Flux

Flux, Zeiss

Outliers

Assess some of the observations outside of expected limits.

eyeid	ptid	female	eye_imaged	image_area	image_size	image_method	exam_date	time_stamp	v_diameter	v_area_density	v_skeleton_density	v_perimeter_index	v_complexity_index	flow_impair_zone	flux	signal_strength	zeiss_v_area_density	zeiss_flux	age	htn_dx	dm_dx	nro_dx	dr_dx	glc_sus_dx	glc_dx	surg_proc_glaucoma	surg_proc_catextract	glc_dx_bin	dr_dx_bin	order_obs	day_one	diff_date	ind_n_day	n_exams
27	1787031	Female	Right	ONH	6x6	Angiography	2018-10-10	2018-10-10 10:45:55	25.08622	0.3086875	0.1230506	0.2841450	4433.711	7.5607	0.10895244	8	0.380	0.288	52	No	No	No	None	Severe	None	None	Yes	Normal	No	1	2018-10-10	0 days	1	3
27	1787031	Female	Right	ONH	6x6	Angiography	2019-04-10	2019-04-10 11:47:25	25.55900	0.2113727	0.0826999	0.1941514	3261.600	12.6711	0.06827361	9	0.386	0.286	52	No	No	No	None	Severe	None	None	Yes	Normal	No	2	2018-10-10	182 days	1	3
27	1787031	Female	Right	ONH	6x6	Angiography	2019-05-08	2019-05-08 11:09:31	25.46251	0.2898345	0.1138279	0.2666664	4229.638	8.1466	0.10083495	7	0.380	0.278	52	No	No	No	None	Severe	None	None	Yes	Normal	No	3	2018-10-10	210 days	1	3

Repeatability

Repeatability of OCTA Variables

Calculate the within-subject standard deviation \((S_w)\), the within-subject coefficient of repeatability \((CR_w)\), and the within-subject coefficient of variation \((CV_w)\) for a given measurement variable \(x\).

Table 2: Repeatability

Table 2 shows repeatability estimates of vessel density measurements.

This table was modeled after Table 2 in Venugopal (2018).²

Intra-Visit Repeatability

Normal

OCTA Variable	n_measure	µ	Sw	Sw 95% CL	CRw	CRw 95% CL	CVw	CVw 95% CL
Vessel Diameter (units?)	73	24.990	0.260	(0.239, 0.282)	0.721	(0.661, 0.781)	1.041	(0.955, 1.127)
Vessel Area Density (units?)	73	0.364	0.023	(0.023, 0.023)	0.064	(0.063, 0.065)	6.313	(6.194, 6.433)
Vessel Area Density (units?), Zeiss Commercial	73	0.433	0.006	(0.006, 0.006)	0.017	(0.017, 0.017)	1.446	(1.443, 1.448)
Vessel Skeleton Density (units?)	73	0.146	0.010	(0.010, 0.010)	0.027	(0.027, 0.027)	6.627	(6.576, 6.677)
Vessel Perimeter Index (units?)	73	0.332	0.021	(0.020, 0.021)	0.057	(0.056, 0.058)	6.222	(6.111, 6.332)
Vessel Complexity Index (units?)	73	5439.739	326.913	(-79431.818, 80085.644)	906.157	(-220173.767, 221986.080)	6.010	(-1460.214, 1472.233)
Flow Impairment Zone (units?)	73	5.169	1.231	(0.081, 2.382)	3.413	(0.224, 6.602)	23.820	(1.566, 46.074)
Flux (units?)	73	0.131	0.009	(0.009, 0.009)	0.026	(0.026, 0.026)	7.143	(7.094, 7.192)
Flux (units?), Zeiss Commercial	73	0.357	0.006	(0.006, 0.006)	0.017	(0.017, 0.017)	1.743	(1.740, 1.745)

Glaucoma

OCTA Variable	n_measure	µ	Sw	Sw 95% CL	CRw	CRw 95% CL	CVw	CVw 95% CL
Vessel Diameter (units?)	141	26.291	0.365	(0.335, 0.395)	1.012	(0.928, 1.096)	1.389	(1.273, 1.504)
Vessel Area Density (units?)	141	0.285	0.014	(0.014, 0.014)	0.038	(0.038, 0.038)	4.833	(4.820, 4.846)
Vessel Area Density (units?), Zeiss Commercial	141	0.398	0.007	(0.007, 0.007)	0.019	(0.019, 0.019)	1.763	(1.759, 1.766)
Vessel Skeleton Density (units?)	141	0.111	0.006	(0.006, 0.006)	0.017	(0.017, 0.017)	5.463	(5.456, 5.470)
Vessel Perimeter Index (units?)	141	0.262	0.013	(0.013, 0.013)	0.035	(0.035, 0.035)	4.797	(4.785, 4.808)
Vessel Complexity Index (units?)	141	4305.536	208.586	(-8024.602, 8441.774)	578.171	(-22243.061, 23399.403)	4.845	(-186.379, 196.068)
Flow Impairment Zone (units?)	141	8.815	0.978	(0.806, 1.150)	2.710	(2.233, 3.188)	11.093	(9.140, 13.047)
Flux (units?)	141	0.100	0.005	(0.005, 0.005)	0.015	(0.015, 0.015)	5.328	(5.323, 5.334)
Flux (units?), Zeiss Commercial	141	0.324	0.007	(0.007, 0.007)	0.018	(0.018, 0.018)	2.047	(2.045, 2.050)

Inter-Visit Repeatability

Restrict to repeat measurements within the last 6 months.

Normal, 180 Days

There are way too few measurements for this table to be reliable.

OCTA Variable	n_measure	µ	Sw	Sw 95% CL	CRw	CRw 95% CL	CVw	CVw 95% CL
Vessel Diameter (units?)	12	24.782	0.371	(0.221, 0.522)	1.030	(0.612, 1.447)	1.499	(0.891, 2.107)
Vessel Area Density (units?)	12	0.360	0.017	(0.017, 0.017)	0.047	(0.047, 0.048)	4.726	(4.672, 4.779)
Vessel Area Density (units?), Zeiss Commercial	12	0.430	0.007	(0.007, 0.007)	0.019	(0.019, 0.019)	1.585	(1.579, 1.591)
Vessel Skeleton Density (units?)	12	0.146	0.006	(0.006, 0.006)	0.018	(0.017, 0.018)	4.345	(4.325, 4.366)
Vessel Perimeter Index (units?)	12	0.330	0.014	(0.014, 0.014)	0.039	(0.039, 0.039)	4.250	(4.211, 4.289)
Vessel Complexity Index (units?)	12	5386.578	188.001	(-22020.906, 22396.909)	521.113	(-61038.837, 62081.064)	3.490	(-408.811, 415.791)
Flow Impairment Zone (units?)	12	5.097	0.439	(0.270, 0.607)	1.216	(0.748, 1.684)	8.608	(5.297, 11.920)
Flux (units?)	12	0.129	0.009	(0.009, 0.009)	0.024	(0.024, 0.025)	6.838	(6.792, 6.885)
Flux (units?), Zeiss Commercial	12	0.357	0.010	(0.010, 0.010)	0.028	(0.027, 0.028)	2.803	(2.776, 2.829)

Glaucoma, 180 Days

OCTA Variable	n_measure	µ	Sw	Sw 95% CL	CRw	CRw 95% CL	CVw	CVw 95% CL
Vessel Diameter (units?)	91	27.004	0.612	(0.514, 0.710)	1.695	(1.424, 1.967)	2.265	(1.902, 2.628)
Vessel Area Density (units?)	91	0.255	0.021	(0.021, 0.021)	0.058	(0.057, 0.058)	8.157	(8.121, 8.192)
Vessel Area Density (units?), Zeiss Commercial	91	0.389	0.009	(0.009, 0.009)	0.026	(0.026, 0.026)	2.409	(2.404, 2.415)
Vessel Skeleton Density (units?)	91	0.097	0.009	(0.009, 0.009)	0.025	(0.025, 0.025)	9.407	(9.390, 9.425)
Vessel Perimeter Index (units?)	91	0.234	0.019	(0.019, 0.019)	0.053	(0.053, 0.054)	8.213	(8.180, 8.246)
Vessel Complexity Index (units?)	91	3841.477	301.043	(-18685.964, 19288.049)	834.448	(-51794.849, 53463.744)	7.837	(-486.426, 502.100)
Flow Impairment Zone (units?)	91	10.365	1.794	(0.824, 2.763)	4.972	(2.285, 7.659)	17.304	(7.952, 26.656)
Flux (units?)	91	0.089	0.009	(0.009, 0.009)	0.024	(0.024, 0.024)	9.701	(9.681, 9.720)
Flux (units?), Zeiss Commercial	91	0.313	0.011	(0.011, 0.011)	0.029	(0.029, 0.030)	3.394	(3.386, 3.402)

Calculating Repeatability Statistics

Explanations of repeatability statistic calculations

Within-Subject Mean \((\mu_w)\)

\(\mu_w\) is the average of all measurements of one eye for a given individual. The overall mean \((\mu)\) is calculated by taking the mean of \(\mu_w\) across the dataset. The overall mean is included in the final table for reference.

\[ \mu_w = \frac{1}{M}\sum_{i=1}^{M} x_i \]

\[\text{where } M \text{ is the number of measurements per eye,}\]

Within-Subject Standard Deviation \((S_w)\)

Find \(S_w\) by first calculating the variance of the measurements per individual eye. The equation below is general for any number of measurements; in this study there are at least 2 measurements per eye, but sometimes more.

\[ \sigma^2_{measurement} = \frac{1}{M}\sum_{i=1}^{M} (x_i - \mu_w)^2 \]

\[ \text{where } M \text{ is the number of measurements per eye} \]

To calculate \(S_w\), average of the variance of measurements \((\sigma^2_{measurement})\) for all eyes measured, then take the square root.

\[ Sw = \sqrt{\frac{1}{N}\sum_{i=1}^{N}\sigma^2_{measurement,i}} \]

\[ \text{where } N \text{ is the number of eyes measured} \]

Within-Subject Coefficient of Repeatability \((CR_w)\)

The \(CR_w\) provides the uncertainty of repeated measures.

\[ CRw = \sqrt2 * 1.96 * Sw \]

The \(CR_w\) can be interpreted as:

“The difference between two measurements for the same subject is expected to be less than [\(CR_w\)] for 95% of pairs of observations.”³

Within-Subject Coefficient of Variation \((CV_w)\)

\[ CVw = 100 *\frac{Sw}{\mu} \]

Intraclass Correlation Coefficient, Unadjusted

Intraclass Correlation Coefficient (ICC) is scored using the ICCest() function from the ICC package. This function “estimates the ICC and confidence intervals using the variance components from a one-way ANOVA.”

This is a preliminary result, as we want to calculate the ICC after adjusting for participant, eye (left or right), signal strength, and clinical and other variables. This will be done later in the mixed effects regression models.

Intra-Visit ICC, Unadjusted

Normal

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	29	0.859	(0.748, 0.927)	2.514	0.07	0.425
Vessel Area Density (units?)	29	0.828	(0.698, 0.911)	2.514	0.00	0.002
Vessel Area Density (units?), Zeiss Commercial	29	0.939	(0.886, 0.969)	2.514	0.00	0.001
Vessel Skeleton Density (units?)	29	0.849	(0.731, 0.922)	2.514	0.00	0.000
Vessel Perimeter Index (units?)	29	0.826	(0.694, 0.909)	2.514	0.00	0.002
Vessel Complexity Index (units?)	29	0.827	(0.696, 0.910)	2.514	97549.90	467168.246
Flow Impairment Zone (units?)	29	0.821	(0.686, 0.907)	2.514	1.36	6.233
Flux (units?)	29	0.857	(0.745, 0.926)	2.514	0.00	0.000
Flux (units?), Zeiss Commercial	29	0.954	(0.914, 0.977)	2.514	0.00	0.001

Glaucoma

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	59	0.961	(0.939, 0.976)	2.388	0.134	3.341
Vessel Area Density (units?)	59	0.973	(0.957, 0.983)	2.388	0.000	0.007
Vessel Area Density (units?), Zeiss Commercial	59	0.965	(0.945, 0.978)	2.388	0.000	0.001
Vessel Skeleton Density (units?)	59	0.974	(0.959, 0.984)	2.388	0.000	0.001
Vessel Perimeter Index (units?)	59	0.973	(0.957, 0.983)	2.388	0.000	0.006
Vessel Complexity Index (units?)	59	0.971	(0.954, 0.982)	2.388	42493.815	1436037.435
Flow Impairment Zone (units?)	59	0.949	(0.920, 0.969)	2.388	0.959	17.992
Flux (units?)	59	0.973	(0.957, 0.983)	2.388	0.000	0.001
Flux (units?), Zeiss Commercial	59	0.969	(0.951, 0.981)	2.388	0.000	0.001

Inter-Visit ICC, Unadjusted

Inter-Visit ICC Normal

Normal, All

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	29	0.753	(0.574, 0.870)	2.373	0.113	0.345
Vessel Area Density (units?)	29	0.782	(0.619, 0.886)	2.373	0.001	0.002
Vessel Area Density (units?), Zeiss Commercial	29	0.898	(0.811, 0.949)	2.373	0.000	0.001
Vessel Skeleton Density (units?)	29	0.804	(0.653, 0.898)	2.373	0.000	0.000
Vessel Perimeter Index (units?)	29	0.809	(0.662, 0.901)	2.373	0.000	0.002
Vessel Complexity Index (units?)	29	0.793	(0.636, 0.893)	2.373	128746.397	494377.918
Flow Impairment Zone (units?)	29	0.786	(0.625, 0.889)	2.373	1.839	6.765
Flux (units?)	29	0.750	(0.570, 0.868)	2.373	0.000	0.000
Flux (units?), Zeiss Commercial	29	0.875	(0.771, 0.937)	2.373	0.000	0.001

Normal, 365 Days

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	15	0.729	(0.391, 0.898)	2.065	0.139	0.374
Vessel Area Density (units?)	15	0.825	(0.576, 0.936)	2.065	0.001	0.003
Vessel Area Density (units?), Zeiss Commercial	15	0.883	(0.704, 0.958)	2.065	0.000	0.000
Vessel Skeleton Density (units?)	15	0.838	(0.604, 0.941)	2.065	0.000	0.001
Vessel Perimeter Index (units?)	15	0.821	(0.568, 0.935)	2.065	0.000	0.002
Vessel Complexity Index (units?)	15	0.873	(0.681, 0.955)	2.065	88763.449	612576.682
Flow Impairment Zone (units?)	15	0.848	(0.624, 0.945)	2.065	1.607	8.934
Flux (units?)	15	0.818	(0.561, 0.933)	2.065	0.000	0.000
Flux (units?), Zeiss Commercial	15	0.935	(0.828, 0.977)	2.065	0.000	0.001

Normal, 180 Days

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	6	0.795	(0.188, 0.968)	2	0.138	0.536
Vessel Area Density (units?)	6	0.891	(0.488, 0.984)	2	0.000	0.002
Vessel Area Density (units?), Zeiss Commercial	6	0.844	(0.328, 0.976)	2	0.000	0.000
Vessel Skeleton Density (units?)	6	0.930	(0.643, 0.990)	2	0.000	0.001
Vessel Perimeter Index (units?)	6	0.909	(0.558, 0.986)	2	0.000	0.002
Vessel Complexity Index (units?)	6	0.932	(0.654, 0.990)	2	35344.536	487622.692
Flow Impairment Zone (units?)	6	0.982	(0.898, 0.997)	2	0.192	10.587
Flux (units?)	6	0.841	(0.319, 0.976)	2	0.000	0.000
Flux (units?), Zeiss Commercial	6	0.836	(0.303, 0.975)	2	0.000	0.001

Inter-Visit ICC Glaucoma

Glaucoma

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	59	0.918	(0.876, 0.947)	2.908	0.277	3.089
Vessel Area Density (units?)	59	0.930	(0.894, 0.955)	2.908	0.000	0.006
Vessel Area Density (units?), Zeiss Commercial	59	0.943	(0.914, 0.964)	2.908	0.000	0.001
Vessel Skeleton Density (units?)	59	0.930	(0.895, 0.956)	2.908	0.000	0.001
Vessel Perimeter Index (units?)	59	0.930	(0.894, 0.955)	2.908	0.000	0.005
Vessel Complexity Index (units?)	59	0.927	(0.890, 0.953)	2.908	96256.921	1215869.381
Flow Impairment Zone (units?)	59	0.856	(0.788, 0.906)	2.908	2.568	15.243
Flux (units?)	59	0.921	(0.882, 0.950)	2.908	0.000	0.001
Flux (units?), Zeiss Commercial	59	0.917	(0.876, 0.947)	2.908	0.000	0.001

Glaucoma, 365 Days

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	48	0.917	(0.867, 0.950)	2.597	0.335	3.696
Vessel Area Density (units?)	48	0.941	(0.905, 0.965)	2.597	0.000	0.006
Vessel Area Density (units?), Zeiss Commercial	48	0.946	(0.913, 0.968)	2.597	0.000	0.001
Vessel Skeleton Density (units?)	48	0.943	(0.907, 0.966)	2.597	0.000	0.001
Vessel Perimeter Index (units?)	48	0.940	(0.904, 0.965)	2.597	0.000	0.005
Vessel Complexity Index (units?)	48	0.941	(0.904, 0.965)	2.597	87527.689	1390759.850
Flow Impairment Zone (units?)	48	0.864	(0.787, 0.917)	2.597	2.774	17.575
Flux (units?)	48	0.931	(0.890, 0.959)	2.597	0.000	0.001
Flux (units?), Zeiss Commercial	48	0.928	(0.884, 0.957)	2.597	0.000	0.001

Glaucoma, 180 Days

OCTA Variable	n_people	ICC	ICC 95% CL	k	varw	vara
Vessel Diameter (units?)	37	0.918	(0.859, 0.955)	2.454	0.389	4.384
Vessel Area Density (units?)	37	0.946	(0.905, 0.971)	2.454	0.000	0.008
Vessel Area Density (units?), Zeiss Commercial	37	0.948	(0.910, 0.972)	2.454	0.000	0.002
Vessel Skeleton Density (units?)	37	0.946	(0.906, 0.971)	2.454	0.000	0.001
Vessel Perimeter Index (units?)	37	0.945	(0.903, 0.970)	2.454	0.000	0.006
Vessel Complexity Index (units?)	37	0.946	(0.905, 0.971)	2.454	93426.165	1630910.630
Flow Impairment Zone (units?)	37	0.871	(0.781, 0.928)	2.454	3.175	21.394
Flux (units?)	37	0.936	(0.889, 0.965)	2.454	0.000	0.001
Flux (units?), Zeiss Commercial	37	0.936	(0.889, 0.965)	2.454	0.000	0.002

Calculating ICC Statistic

\[ \text{ICC} = \frac{\text{between}}{\text{between} + \text{within}}\]

“In the output for the random effects model ‘SID’ is the estimated between-group variance, which in the ICCest output is called ‘vara’ (variance among groups). The within-group variance is ‘Residual’ in the random effects model output, while in ICCest it’s called ‘varw.’ You can see that the values for those two variances match perfectly in the two different outputs. The ICC is defined as between/(between+within). That calculation is not part of the random effects model output, but it is part of the ICCest output, along with confidence intervals for the ICC estimate. The ICC are really high for intravisit (around 97%) and a little lower for intervisit (about 94%). If we were to calculate these ICC for glaucoma and controls separately I think they would be significantly lower because of the reduced between-group variability.” - BB 9/4/2019

ARVO Abstract Tables

Table 1: Participants

There were:

• 73 measurements of 29 patients without glaucoma

• 141 measurements of 59 patients with glaucoma

	None	Glaucoma	p	test
n	29	59
Sociodemographic Variables
age (mean (SD))	52.31 (17.08)	60.15 (15.12)	0.031
Sex = Male (%)	13 (44.8)	30 ( 50.8)	0.761
Systemic Clinical Variables
Hypertension Diagnosis = Yes (%)	9 (31.0)	21 ( 35.6)	0.853
Diabetes Mellitus Diagnosis = Yes (%)	2 ( 6.9)	11 ( 18.6)	0.254
Optic Neuropathy = Yes (%)	1 ( 3.4)	0 ( 0.0)	0.715
Ophthalmic Clinical Variables
Glaucoma Diagnosis, Binary = Glaucoma (%)	0 ( 0.0)	59 (100.0)	<0.001
Diabetic Retinopathy Diagnosis, Binary = Yes (%)	0 ( 0.0)	4 ( 6.8)	0.373
Ophthalmic Surgical Variables
Surgical Procedure for Cataract Extraction = Yes (%)	5 (17.2)	13 ( 22.0)	0.808
Surgical Procedure for Glaucoma (%)			0.004
None	25 (86.2)	31 ( 52.5)
Laser	4 (13.8)	8 ( 13.6)
Minimally Invasive Glaucoma Surgery	0 ( 0.0)	1 ( 1.7)
Filter	0 ( 0.0)	19 ( 32.2)
OCT Angiography Signal Strength
signal_strength (mean (SD))	9.29 (0.71)	9.11 (0.75)	0.278

Table 2: Repeatability Statistics

OCTA Variable	Mean		ICC				Sw				CRw				CVw
	None	Glaucoma	None		Glaucoma		None		Glaucoma		None		Glaucoma		None		Glaucoma
var_name_normal	µ_normal	µ_glc	ICC_pretty_normal	ICC_pretty_cl_normal	ICC_pretty_glc	ICC_pretty_cl_glc	Sw_pretty_normal	Sw_pretty_cl_normal	Sw_pretty_glc	Sw_pretty_cl_glc	CRw_pretty_normal	CRw_pretty_cl_normal	CRw_pretty_glc	CRw_pretty_cl_glc	CVw_pretty_normal	CVw_pretty_cl_normal	CVw_pretty_glc	CVw_pretty_cl_glc
Vessel Area Density (units?)	0.364	0.285	0.828	(0.698, 0.911)	0.973	(0.957, 0.983)	0.023	(0.023, 0.023)	0.014	(0.014, 0.014)	0.064	(0.063, 0.065)	0.038	(0.038, 0.038)	6.313	(6.194, 6.433)	4.833	(4.820, 4.846)
Vessel Area Density (units?), Zeiss Commercial	0.433	0.398	0.939	(0.886, 0.969)	0.965	(0.945, 0.978)	0.006	(0.006, 0.006)	0.007	(0.007, 0.007)	0.017	(0.017, 0.017)	0.019	(0.019, 0.019)	1.446	(1.443, 1.448)	1.763	(1.759, 1.766)
Vessel Skeleton Density (units?)	0.146	0.111	0.849	(0.731, 0.922)	0.974	(0.959, 0.984)	0.010	(0.010, 0.010)	0.006	(0.006, 0.006)	0.027	(0.027, 0.027)	0.017	(0.017, 0.017)	6.627	(6.576, 6.677)	5.463	(5.456, 5.470)
Flux (units?)	0.131	0.100	0.857	(0.745, 0.926)	0.973	(0.957, 0.983)	0.009	(0.009, 0.009)	0.005	(0.005, 0.005)	0.026	(0.026, 0.026)	0.015	(0.015, 0.015)	7.143	(7.094, 7.192)	5.328	(5.323, 5.334)
Flux (units?), Zeiss Commercial	0.357	0.324	0.954	(0.914, 0.977)	0.969	(0.951, 0.981)	0.006	(0.006, 0.006)	0.007	(0.007, 0.007)	0.017	(0.017, 0.017)	0.018	(0.018, 0.018)	1.743	(1.740, 1.745)	2.047	(2.045, 2.050)

Mixed Effects Regression Models

Model Fitting

“For the markdown file let’s just spit out all the output from the random effects model and the iccest command so we can show it to Grace. Eventually we can put the ICC in the table with the other repeatability measures you have already tabled but I don’t think we need it for the meeting.” - BB 9/4/2019

## Linear mixed model fit by REML ['lmerMod']
## Formula: v_area_density ~ (1 | ptid)
##    Data: data_octa_global_intra
## 
## REML criterion at convergence: -776.7
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.5205 -0.4207 -0.0321  0.4433  4.4693 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  ptid     (Intercept) 0.006630 0.08143 
##  Residual             0.000288 0.01697 
## Number of obs: 214, groups:  ptid, 88
## 
## Fixed effects:
##             Estimate Std. Error t value
## (Intercept)  0.30807    0.00876   35.17

“I also added a model to test whether SS influences reliability. The reference level is ‘10’ now, which is what we want, and you can see that SS is highly associated with VAD, meaning it affects reliability.” - BB 9/4/2019

## Linear mixed model fit by REML ['lmerMod']
## Formula: v_area_density ~ signal_strength + (1 | ptid)
##    Data: data_octa_global_intra
## 
## REML criterion at convergence: -790
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -5.1871 -0.4090 -0.0265  0.4069  5.1579 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.
##  ptid     (Intercept) 0.0059998 0.07746 
##  Residual             0.0002587 0.01608 
## Number of obs: 214, groups:  ptid, 88
## 
## Fixed effects:
##                 Estimate Std. Error t value
## (Intercept)     0.181217   0.026988   6.715
## signal_strength 0.013832   0.002799   4.942
## 
## Correlation of Fixed Effects:
##             (Intr)
## sgnl_strngt -0.951

References

1. Venugopal JP, Rao HL, Weinreb RN, et al. Repeatability and comparability of peripapillary vessel density measurements of high-density and non-high-density optical coherence tomography angiography scans in normal and glaucoma eyes. The British Journal of Ophthalmology 2019;103:949–954.

2. Venugopal JP, Rao HL, Weinreb RN, et al. Repeatability of vessel density measurements of optical coherence tomography angiography in normal and glaucoma eyes. The British Journal of Ophthalmology 2018;102:352–357.

3. Bland JM, Altman DG. Statistics Notes: Measurement error. BMJ : British Medical Journal 1996;313:744.